Situational awareness for teleoperation of a remote vehicle

ABSTRACT

A method for improving situational awareness for teleoperation of a remote vehicle by creating a 3D map display of an area around the remote vehicle comprises: receiving an original image from a stereo vision camera and utilizing the original image to perform visual odometry to determine the x, y, z, roll, pitch, and yaw for the original image; applying a fill-in algorithm to the original image to fill in an estimated depth for areas of the original image for which no depth data is available, which creates an enhanced depth image; combining the enhanced depth image with the x, y, z, roll, pitch, and yaw for the original image to create the 3D map display of the area around the remote vehicle; and displaying the 3D map display on an operator control unit used to control the remote vehicle.

RELATED APPLICATION INFORMATION

This application is a continuation application of, and claims priorityto, U.S. Application No. 13/030,120, filed Feb. 17, 2011 which claimsthe benefit, under 35 U.S.C. §119(e), of U.S. Provisional ApplicationNo. 61/305,384, filed Feb. 17, 2010, the entire contents of each ofwhich are incorporated by reference herein.

GOVERNMENT SUPPORT

This invention was made with government support under Contract No.N00164-05-DD647 awarded by the Department of the Navy. The governmenthas certain rights in the invention.

INTRODUCTION

The present teachings relate to a system and method for improvingsituational awareness for teleoperating a remote vehicle. The presentteachings more particularly provide a situational awareness payload fora remote vehicle, and user interface modifications to an operatorcontrol unit. The operator control unit provides a 3D map display of thearea around the remote vehicle constructed using stereo vision data,visual odometry, and heuristics to fill in depth data in areas that arelacking in texture.

BACKGROUND

Currently, Explosive Ordnance Disposal (EOD) and similar remote vehicleapplications involve time-consuming and complex maneuvers that requirethe operator to expend significant cognitive attention in commanding andcontrolling the remote vehicle. In addition to controlling the remotevehicle, the operator has to consider the surroundings of the remotevehicle down range. This attention overload can leave the operatordangerously unaware of events in his own immediate environment.

Further, the operator is required to control every motion of the remotevehicle using limited sensory information and situational awareness,primarily provided by one or more video streams from independentmonocular cameras. This required level of detailed control, coupled witha lack of situational awareness, can result in increased missionexecution time. Increased mission execution time can cause increasedtime on target in areas of elevated risk to the operator and anysupporting personnel.

SUMMARY

The present teachings provide a method for improving situationalawareness for teleoperation of a remote vehicle by creating a 3D mapdisplay of an area around the remote vehicle, comprising: receiving anoriginal image from a stereo vision camera and utilizing the originalimage to perform visual odometry to determine the x, y, z, roll, pitch,and yaw for the original image; applying a fill-in algorithm to theoriginal image to fill in an estimated depth for areas of the originalimage for which no depth data is available, which creates an enhanceddepth image; combining the enhanced depth image with the x, y, z, roll,pitch, and yaw for the original image to create the 3D map display ofthe area around the remote vehicle; and displaying the 3D map display onan operator control unit used to control the remote vehicle

The present teachings also provide a system for providing improvedsituational awareness for teleoperation of a remote vehicle having achassis, the system comprising: a remote vehicle having a range datapayload and a computational payload, the computation payload providing avisual odometry map construction for a 3D image; and an operator controlunit including a display screen displaying the 3D image of an areaaround the remote vehicle.

The present teachings further provide a system for providing improvedsituational awareness for teleoperation of a remote vehicle having achassis, the system comprising: a remote vehicle having a range datapayload and a computational payload, and a manipulator arm payload, thecomputation payload providing a visual odometry map construction for a3D image; and an operator control unit including a display screendisplaying the 3D image of an area around the remote vehicle, the 3Dimage being constructed from raw stereo vision data, visual odometry,fill-in algorithms, and plane fitting.

Additional objects and advantages of the present teachings will be setforth in part in the description which follows, and in part will beobvious from the description, or may be learned by practice of thepresent teachings. The objects and advantages of the present teachingswill be realized and attained by means of the elements and combinationsparticularly pointed out in the appended claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the present teachings, as claimed.

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate several embodiments of thepresent teachings and together with the description, serve to explainthe principles of the present teachings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of exemplary system architecture inaccordance with the present teachings.

FIG. 2A illustrates an exemplary system in accordance with the presentteachings, including a remote vehicle having payloads suitable foracquiring and processing data for use in accordance with the presentteachings and an operator control unit for providing a 3D display inaccordance with the present teachings.

FIG. 2B illustrates an exemplary remote vehicle having payloads suitablefor acquiring and processing data for use in accordance with the presentteachings.

FIG. 2C illustrates an exemplary range data payload for use inaccordance with the present teachings.

FIG. 2D illustrates an exemplary computational payload for use inaccordance with the present teachings.

FIG. 3 is a flow chart setting forth an exemplary visual odometryroutine.

FIG. 4 is a flow chart setting forth an exemplary embodiment of afill-in algorithm.

FIG. 5 is a flow chart setting forth an exemplary embodiment of a planefitting algorithm for the fill-in algorithm of FIG. 4.

FIGS. 6A, 7A, 8A, and 9A illustrate an operator control unit screen withan exemplary embodiment of a 3D map display created in accordance withthe present teachings, using raw stereo vision camera data.

FIGS. 6B, 7B, 8B, and 9B illustrate an operator control unit screen withan exemplary embodiment of a 3D map display created in accordance withthe present teachings, using stereo vision camera data in combinationwith visual odometry, fill in algorithms, and plane fitting.

DESCRIPTION

Reference will now be made in detail to embodiments of the presentteachings, examples of which are illustrated in the accompanyingdrawings.

The present teachings contemplate improving situational awareness andreducing time on target for missions such as EOD missions made byoperators utilizing a remote vehicle such as a small unmanned groundvehicle, for example an iRobot® Packbot® and more specifically aniRobot® Packbot® MTRS EOD robot. The remote vehicle can be equipped witha situational awareness payload, the one or more components of which areexplained in more detail below.

In accordance with certain embodiments of the present teachings, theproposed situational awareness payload is modular, and is capable ofbeing connected to one or more existing payload ports of the remotevehicle, requiring minimal modifications to the base remote vehiclesystem.

The associated operator control unit can include appropriate userinterface modifications to accommodate and display data received fromthe situational awareness payload. The user interface modifications canprovide enhanced awareness by affording the operator a virtualized 3Dview of the remote vehicle platform, an optional manipulator payload ofthe remote vehicle, and the local environment surrounding the remotevehicle. In addition to providing the operator with a detailed view ofthe remote vehicle and its spatial position within its environment, theuser interface can facilitate command and control of the remote vehicleand its manipulator through a point-and-click-based interface with thedisplayed 3D remote vehicle and the 3D map of its environment. Incertain embodiments of the present teachings wherein the operator cansee the remote vehicle in its actual position on the OCU display screen,the operator may be able to direct the remote vehicle intuitively,seeing the onscreen model mimic the actions of its real worldcounterpart.

FIG. 1 is a schematic diagram of exemplary system architecture inaccordance with the present teachings. As can be seen, a remote vehiclechassis can have payloads such as a computational payload (e.g., aNavigator payload), a rnge data payload (e.g., a SEER stereo visionpayload), and manipulator arm mounted thereon. A typical Navigatorpayload provides additional computation capabilities and sensors neededto process and interpret sensor data provided by the SEER payload. TheNavigator payload is illustrated in FIG. 2C and can comprise, forexample, a computer, for example an Intel Core Duo 1.2 GHz processor, 1GB of RAM, and an 8 GB solid state flash memory hard drive, a six-axismicro-electro-mechanical system (MEMS) inertial measurement unit (IMU)(e.g., a Microstrain 3DM-GX1), and optionally a GPS receiver (e.g., aUblox Antaris 4 GPS receiver) and a magnetometer. The GPS receiver cantypically determine the remote vehicle's position accurately to withinapproximately 2-4 meters. The IMU can typically determine theorientation of the remote vehicle and can have a drift rate of less than1° per minute. The GPS and IMU therefore provide information regardingwhere the robot is in its environment. As will be understood by thoseskilled in the art, reference herein to a Navigator payload includesother payloads having the same or substantially the same componentsand/or capabilities.

A typical SEER payload is illustrated in FIG. 2B and provides rangingdata that can be used to provide the operator with an improvedunderstanding of the spatial layout of the environment immediatelyaround the remote vehicle, and can comprises a stereo vision camera, astereo depth comparison chip, and a small processor that can be used topackage and send data. The stereo vision camera can be, for example,mounted under the manipulator arm or otherwise mounted to obtain anunobstructed view of the remote vehicle's environment, and can provide,for example, 500×312 high-resolution 3D range data at 30 frames persecond with a maximum range of 50 meters and with a range accuracy of0-3% over a range of 0-5 meters. As will be understood by those skilledin the art, reference herein to a SEER payload includes other payloadshaving the same or substantially the same components and/orcapabilities.

The manipulator arm payload can comprise, for example, a gripper armpayload having a gripper and a camera mounted thereon and trainable onthe gripper. As will be understood by those skilled in the art, asituational awareness payload can comprise one or more payloads havingthe necessary components and/or capabilities available in theabove-described Navigator and SEER payloads, and can optionally includea manipulator arm payload. The manipulator arm can be provided as partof the situational awareness payload, as a separate payload, or may notbe provided on the remote vehicle.

An exemplary system in accordance with the present teaching isillustrated in FIG. 2A to comprise a small unmanned ground vehicle(i.e., an iRobot® Packbot®) including a computation payload (i.e., aNavigator payload), a range data payload (not shown), and a manipulatorarm shown to have a gripper thereon, and an operator control unitcomprising a display screen with a 3D map display in one of itsquadrants.

An exemplary embodiment of a remote vehicle having the above-listedpayload components and capabilities is illustrated in FIG. 2B. As shownin FIG. 1, the remote vehicle can communicate with the attached payloadsvia, for example, an Ethernet connection. As will be readily understood,other known methods of communication can additionally or alternativelybe employed. The remote vehicle chassis can include a power supply suchas, for example, one or more batteries or fuel cells, and can providepower for each of the payloads. Alternatively, some or all of thepayloads can include their own power supply such as a battery or fuelcell.

In the embodiment of FIG. 1, the remote vehicle chassis communicateswith the Navigator payload via Ethernet and provides power to theNavigator payload. The chassis also communicates with the SEER payloadvia Ethernet and provides power to the SEER payload. In addition, in theillustrated embodiment wherein the manipulator arm payload plugs intothe SEER payload, the chassis provides power to the manipulator armpayload via the SEER payload. Further, analog video from a camera on themanipulator arm can be passed to the chassis via the SEER payload, and aproprietary protocol for providing operator instructions to themanipulator arm payload, such as FarNet, can be passed to themanipulator arm via instructions sent from the operator control unit tothe remote vehicle chassis and from the remote vehicle chassis to themanipulator arm payload via the SEER payload.

In accordance with various embodiments, a JAUS 3.4 compatible messagingcapability can be provided for communications between the OCU and thepayloads. The messages can, for example, communicate range data from theremote vehicle to the OCU and remote vehicle motion data from the remotevehicle's visual odometry system to the OCU.

In accordance with certain embodiments of the present teachings, ananalog camera can be added to the SEER payload or otherwise on theremote vehicle and provide additional visual data for the OCU display,and therefore another analog video line may run to the remote vehiclechassis for the additional camera, for example from the SEER payload.

In certain embodiments, the situational awareness payload and/or itscomponent payloads, for example the Navigator payload, attaches to oneor more modular payload bays on the remote vehicle in a plug-and-playfashion, without requiring modification to the remote vehicle hardwareor software. Similar to the Navigator payload, the SEER payload canattach to one or more modular payload bays on the remote vehicle in aplug-and-play fashion, without requiring modification to the remotevehicle hardware or software. As set forth above, in certain embodimentsof the present teachings, the manipulator arm payload can plug into theSEER payload in a plug-and-play fashion, without requiring modificationto the remote vehicle hardware or software.

The system architecture illustrated in FIG. 1 also includes an operatorcontrol unit (OCU). The OCU can communicate with the remote vehiclechassis via, for example, wireless Ethernet to send commands to theremote vehicle and receive data from the remote vehicle.

The present teachings contemplate providing 3D data in the form of rawstereo data from a stereo vision camera as a panel the OCU. The stereovision camera is part of the stereo data payload and can comprise, forexample, a Tyzx stereo vision camera or other suitable camera providingstereo data. The minimum range of some stereo vision cameras may be toofar for the ideal work environment for the remote vehicle's arm. Tocompensate for minimum range issues, the stereo vision camera can be runin a different mode that trades reduced minimum range for reduced rangeaccuracy in a manner known to those skilled in the art. Such areduced-range mode can, in certain embodiments, be selected dynamically,allowing the user to dynamically choose between improved minimum rangeand improved distance accuracy. Alternatively, the present teachingscontemplate modifying a baseline of the stereo vision camera to reducethe minimum range in a manner known to those skilled in the art, whichcan also reduce the range accuracy.

The raw stereo data can be converted to a 3D map display by constructinga virtual view of the 3D data directly from the stereo vision camera asa panel on the OCU to create an intuitive display of the stereo visiondata useful to operators performing tasks with the remote vehicle.Bandwidth between the remote vehicle and the OCU can constrain theamount of data passed therebetween; however, the perceived quality ofthe display can depend on the resolution of the 3D reconstruction,particularly with regard to texture. Rendering speed for a high-qualitydisplay can also be problematic given the number of polygons andtextures involved in a high-quality display.

The present teachings contemplate providing map construction for the 3Dimage on a computational payload of the remote vehicle, thus minimizingthe amount of data sent to the OCU. Data is processed in thecomputational payload and then sent to the OCU. Communication betweenthe payload and the OCU can be used to send processed map update data,for example at about 1 Hz using about 500 Kbps. Combined with the twovideo streams typically sent from the remote vehicle to the OCU, theremote vehicle therefore may utilize, for example, about 1.25 Mbps ofbandwidth.

In certain embodiments of the present teachings, the 3D map display canbe implemented as a new display quadrant option on an existing OCUscreen, and provide an integrated view of the remote vehicle's currentpose, including flipper and arm positions if applicable, as well as aspatial representation of obstacles around the remote vehicle. Invarious embodiments, the spatial representation can be colorized. Theintegrated 3D display can greatly improve an operator's situationalawareness and enable a faster tempo of operations. In certainembodiments of the present teachings, the display quadrant can beselected for viewing by the operator, and can be quickly shown or hiddendepending on an operator's immediate need. Certain embodiments of thepresent teachings contemplate the additional display quadrant being asupplemental alternative to, rather than a replacement for, currentviews provided by existing remote vehicle OCUs. Thus, impact to existingoperator's can be minimized because current OCU view(s) will still beavailable.

The new display quadrant can comprise a virtualized overhead view of theremote vehicle, showing the position of the remote vehicle and itsmanipulator arm, and the remote vehicle's position with respect toobstacles in the local environment. The virtualized view of the remotevehicle can be positioned on the OCU screen, for example, below aprimary video feed from a camera located at the end of the manipulatorarm, to maximize the correlation between the virtualized viewpoint andthe actual view supplied by an analog video camera mounted on themanipulator arm.

As stated above, the present teachings contemplate structuring the 3Dmap display so that it is directly below the view returned by theprimary video feed from a camera located at the end of the manipulatorarm. The present teachings also contemplate the primary video feed andthe 3D map display being linked together so they always point in thesame direction. However, certain embodiments contemplate the primaryvideo feed and the 3D map display pointing in independent directions.Although having the primary video feed and the 3D map display point inindependent directions may make it difficult for a user to reconcilewhat he or she is seeing on the OCU screen when controlling the remotevehicle, linking the views might make it harder for the user to driveforward.

The local obstacle map for the 3D map display of the OCU can beiteratively constructed using range data provided by the stereo visioncamera combined with odometry information. The range data can bedisplayed using a 3D representation that allows the operator to assessthe height of objects in the environment in order to make navigationdecisions. The present teachings thus contemplate providing color codingto the display based on height above the ground, which can simplify userinterpretation of the displayed 3D map. Objects that present navigationobstacles can, for example, be rendered in red to alert the operator totheir presence and the potential collision hazard. The present teachingsalso contemplate color coding based on distance from the remote vehicle,but color coding for height is preferable. This at-a-glance depiction ofthe surroundings can enable faster and safer teleoperation of the remotevehicle.

The present teachings contemplate providing a depth feedback option forthe 3D map display, as well as the ability to measure distances viaselection on the 3D map display. The display is preferably updated live.

Visual odometry can be utilized in a system of the present teachings forrendering of the 3D map display, providing the x, y, z, roll, pitch, andyaw for images retrieved from the stereo vision camera so that theimages can be accurately added to the 3D map display. Visual odometrycan also be used when creating a virtual remote vehicle for display inaccordance with the present teachings. Visual odometry is the process ofdetermining the position and orientation of a vehicle by analyzingassociated camera images. An advantage of visual odometry is that itprovides an odometry reading based on a wide area of coverage around theremote vehicle due to memory of areas currently out of view of thestereo vision cameras used therefor. Visual odometry can be integratedinto the map construction and thus, for example, can be installed on thecomputational payload. Because visual odometry can drift while a vehicleis stationary and therefore cause a tilted or otherwise inaccurate map,the present teachings contemplate allowing the user to reset the mapwhen such drift is detected or after the vehicle has been stationary foran extended period of time.

Visual odometry algorithms can estimate a six-dimensional position (x,y, z, roll, pitch, and yaw) of the remote vehicle from its initialstarting position using an Iterative Closest Point (ICP) and RandomSample Consensus (RANSAC)—based algorithm. A current exemplary algorithmcan extract key features from a frame, for example approximately 100-500features, and compare them to a reference frame. Matches between thepoints of the two frames are then used to estimate a 4×4 affine matrixtransformation needed to align the two sets of features.

To prevent unwarranted drift, frames with motion beneath a certainthreshold may not affect the cumulative odometry. Similarly, if largeamount of bad matches occur and the estimated motion is larger than themaximum possible motion of the remote vehicle, the frame can be ignored.Such a case can often occur, for example, if large objects, such as aperson, move through the camera's field of view.

Overall, a visual odometry algorithm can produce odometry comparable totracked odometry. Unlike typical encoder odometry, visual odometry maynot be skewed by the slippage that frequently occurs. This can beparticularly evident during turning, where tracked odometry has beenshown to produce errors of up to +−90 degrees in a 180 degree turn. Thevisual odometry algorithm can also provide complete six-dimensionalodometry: x, y, z, roll, pitch, and yaw, whereas tracked odometryprovides only x, y, and yaw.

A flow chart setting forth an exemplary visual odometry routine is shownin FIG. 3. The algorithm takes three points from the stereo visioncamera data and fits the points into a map of the environment or a“scene”. As shown, the visual odometry routine extracts features fromthe image and applies a transform. Thereafter, the algorithm findsmatches from the scene and confirms them. Confirmation of the matchescan include taking the image patch around the point andcross-correlating it. If the cross-correlation is high enough and thereis a mutual consistency, the matches are confirmed. A candidatetransform is them produced by picking a number of the matched pairs,allowing a translation to be determined in x, y, z, roll, pitch, andyaw. The candidate transform is then evaluated to determine whether itcorresponds with other matches from the scene (e.g., how many and howwell they match), and the algorithm determines whether enough candidateshave been confirmed. If so, a best transform is selected. If not, morecandidates are processed as described above. Once the best transform hasbeen selected, the transform can be refined by running a standardminimization scheme. Once the refining step converges (or after apredetermined amount of time or iterations), a reference frame isupdated and stored, and the algorithm is finished with its currentiteration. In addition, after the transform has been refined, it isstored as a best transform and used in subsequent “apply transform”steps.

Holes in the stereo vision data can be an obstacle to the operator'sintuitive understanding of the 3D map display, and thus limit theability of the system to provide improved situational awareness. Missingstereo vision data can result in empty, see-through areas of the 3D mapdisplay that misrepresent the existing solid objects in the environment.The present teachings contemplate intelligently interpolating depthacross the image constructed from the stereo vision data in an effort tofill in or guess at the correct depth for areas having no availablestereo vision data (e.g., for areas lacking surface texture sufficientfor detection by the stereo vision cameras based on the resolution andrange of the camera and lighting conditions).

In accordance with certain embodiments of the present teachings,heuristics can be used to fill in depth (and hence 3D map) data forareas that are lacking in texture. Simple heuristics can look for areaswith no visual texture that had depth information on both sides atroughly consistent depths. The depths between the sides of consistentdepth are filled in by interpolating between the known depths at thesides, providing a more accurate 3D map display.

Building on the simple heuristic above, certain embodiments of thepresent teachings contemplate filling in data missing from the stereovision camera. Data fill-in can be accomplished, for example, bysegmenting the image, fitting planes to the depth data within eachregion, and then using the planes to estimate depth where stereo data ismissing. Segmentation using even an off-the-shelf algorithm (e.g.,Efficient Graph-Based Image Segmentation, Pedro F. Felzenszwalb andDaniel P. Huttenlocher, International Journal of Computer Vision, Volume59, Number 2, September 2004) can result in improved 3D map displayresults; however, an algorithm can be adjusted or created to improve theamount of data availability for the plane fitting step. The presentteachings contemplate allowing the user to turn the fill-in feature onand off as desired.

A schematic diagram of an exemplary embodiment of a fill-in algorithm isillustrated in FIG. 4. As shown, an image from the stereo vision camerais segmented and a plane is fit to the image segment as shown, forexample, in FIG. 5 (discussed below). A depth image is used as input tothe plane fitting step. In accordance with certain embodiments of thepresent teachings, the segmented image includes the image and a list ofthe segment that each pixel belongs to, including depth. If the planefitting step is successful, the algorithm fills in image depth from theplane, stores the depth image, and determines whether all of the imagesegments have been processed. If not, the next segment is plane fittedand processed as described above. The image segmenting algorithm knowshow many segments it produced, and the algorithm loops through thesegments, for example by size, until there are no remaining segmentsfrom the image. Once all of the segments of an image have been processby the fill-in algorithm, the originally-input image is finished and thenext image is segmented and processed accordingly. In certainembodiments, all of the image segments may not be processed, which cansave processing time. The output of the fill-in algorithm is an enhanceddepth image that can be combined with the visual odometry data andprojected into the 3D map display for a more robust representation ofthe remote vehicle's environment.

FIG. 5 illustrates a schematic diagram of an exemplary embodiment of aplane-fitting algorithm for use in the fill-in algorithm of FIG. 4. Asshown, an image segment is dilated (e.g., taking a binary region of theimage segment and dilating it horizontally) and then the dilated imagesegment and a depth image are used to determine location relative to thestereo vision camera and collect 3D points. Thereafter, a Random SampleConsensus (RANSAC)-based algorithm can be used to fit a plane to thepoints. The fit of the plane to the points is then checked (based, forexample, on the number of points that are sufficiently close the plane).If the plane fit is not okay, the system is done and has failed. If theplane fit is okay, the algorithm tries to fit a line to the points. If aline can be fit to the points (i.e., if the line fit is okay), the planeis replaced by a constrained plane and plane fit is checked again. Ifthe second plane fit is not okay, the system is done and has failed. Ifthe second plane fit is okay, the final plane is the second plane andthe system is finished plane fitting for that segment. If a line couldnot be fit to the points of the first plane, the final plane is thefirst plane and the system is finished plane fitting for that segment.

Exemplary versions of 3D map displays created in accordance with thepresent teachings are shown in FIGS. 6A, 6B, 7A, 7B, 8A, 8B, 9A, and 9B.FIGS. 6A, 7A, 8A, and 9A illustrate an exemplary 3D map displayincluding only raw stereo vision data. FIGS. 6B, 7B, 8B, and 9Billustrate a preferred exemplary 3D map display created using stereovision data in combination with visual odometry, a fill-in algorithm,and plane fitting. As will be appreciated, FIGS. 6A, 6B, 7A, 7B, 8A, 8B,9A, and 9B do not adequately identify any color coding that may be addedto the 3D map display. The differences between the “A” figures and the“B” figures show an exemplary amount of improvement to the 3D mapdisplay that can be achieved by combining the raw stereo vision datawith visual odometry, a fill-in algorithm, and plane fitting. FIGS. 6Aand 6B show a video feed of a room with a cabinet, and the associated 3Dmap display thereof. FIGS. 7A and 7B show a video feed of a hallway, andthe associated 3D map display thereof. FIGS. 8A and 8B show a video feedof a storage area with clutter, and the associated 3D map displaythereof. FIGS. 9A and 9B show a video feed of a room with a line ofcrates, and the associated 3D map display thereof.

As shown in the 3D map display of FIGS. 6A, 6B, 7A, 7B, 8A, 8B, 9A, and9B, the 3D map display can include a 3D image of the remote vehicleindicating its position within its environment. In certain embodimentsof the present teachings including such a 3D remote vehicle image, theimage may indicate the exact pose of the remote vehicle and the positionof, for example, an attached gripper arm or camera.

In various embodiments, the displayed 3D image of the remote vehicle ismanipulable by the operator to change the position of the remote vehicleand thus provide a 3D map display for another portion of the remotevehicle's environment. The operator can use the 3D map display to sendmotion commands to the remote vehicle via intuitive, simple, click anddrag operations. For example, in the embodiment of FIG. 6B, the OCU caninclude a user interface allowing the operator to drive the remotevehicle by clicking on one of the arrows surrounding the remote vehicle,which can cause the remote vehicle to move in a direction indicated bythe arrow. For example, the operator can click on the curved arrows tothe right and left of the remote vehicle to pivot the remote vehicle andpan its surroundings. The operator can also click on the arrows to thefront and rear of the remote vehicle to drive the remote vehicle forwardand backward, respectively. Movement of the remote vehicle within itsenvironment changes the video feed and the 3D display to represent itschanging environment.

Using range data provided by the range data payload, the presentteachings contemplate providing a point-and-click-based controlmechanism for positioning the remote vehicle and its manipulator arm.The operator can, for example, use an obstacle map built from the stereovision data to identify navigable paths around the remote vehicle, andthen direct the remote vehicle along a desired path through a point andclick-based interface by clicking on various waypoints designating thedesired path, causing the remote vehicle to execute the motions requiredto traverse that path. This can greatly simplify operator cognitiveload, as compared with current methods of micro-directing remote vehiclemovement, particularly along a tortuous or obstructed path.

In certain embodiments of the present teachings, the same or a similarpoint-and-click-based interface can be used to control the manipulatorarm and its end effector. The 3D map display can allow the operator tosee a three-dimensional representation of the remote vehicle'senvironment and the objects in it, allowing the operator to specifywhere the manipulator arm should be positioned to accomplish the task athand. The operator can click on the manipulator arm on the 3D mapdisplay and drag it to a desired pose or position, and the remotevehicle can execute the motions required to achieve the desired pose orposition.

In certain embodiments of the present teachings, the operator canadditionally or alternatively pan and tilt the virtualized view using anOCU hand controller, for example a joystick on the OCU's hand controller(see FIG. 2A). In accordance with certain embodiments, rotating the viewof the 3D map display allows the operator to see, for example, a 3D viewof what is behind or to the side of the remote vehicle, based on datacollected during a previous traversal of the remote vehicle through itsenvironment. Although the remote vehicle's camera(s) may not be pointedrearward and, in effect, the remote vehicle no longer directly “sees”its environment to the side and rear, the stereo vision camera datainformation can be integrated into a local consistent map, allowing theoperator to evaluate the environment behind the remote vehicle. Thisreal-world situational view can be useful when operators need to operatea remote vehicle in reverse or perform motions in cluttered environmentswhile avoiding potential collisions with surrounding objects orobstacles.

Integrating positioning information provided by visual odometry withdata from the IMU can provide an accurate position estimate for theremote vehicle as it moves, allowing the system to use the stereo visiondata to build an accurate and persistent local map of obstaclessurrounding the remote vehicle. Even when obstacles are no longer shownin a virtualized teleoperation view of the OCU (for example, after theremote vehicle has passed them and they are behind the remote vehicle),their presence can be remembered by a computer of the situationalawareness system, enabling the operator to draw upon that knowledge asrequired.

In accordance with certain embodiments of the present teachings,particularly those implemented on an iRobot® PackBat®, system softwarerunning on the payloads can be written within iRobot's Aware™ 2.0 RobotIntelligence Software.

Other embodiments of the present teachings will be apparent to thoseskilled in the art from consideration of the specification and practiceof the present teachings disclosed herein. It is intended that thespecification and examples be considered as exemplary only, with a truescope and spirit of the present teachings being indicated by thefollowing claims.

What is claimed is:
 1. A system for providing improved situational awareness for teleoperation of a remote vehicle having a chassis, the system comprising: a remote vehicle having a range data payload including an imaging device mounted to obtain an unobstructed view of the remote vehicle's local environment, the imaging device providing data for construction of a 3D image; and an operator control unit including a display screen displaying the 3D image of an area around the remote vehicle, the 3D image being constructed from a raw range data produced by the imaging device.
 2. The system of claim 1, wherein the operator control unit provides an operator with a virtualized 3D view of the remote vehicle and the pose of the remote vehicle within the local environment surrounding the remote vehicle, as well as spatial representations of obstacles around the remote vehicle.
 3. The system of claim 1, wherein the operator control unit further comprises a point-and-click interface on the 3D display of the remote vehicle in the local environment, the point-and-click interface facilitating command and control of the remote vehicle.
 4. The system of claim 3, wherein the point-and click includes click and drag operations wherein the display includes a user interface allowing an operator to drive the remote vehicle by clicking on an arrow to cause the remote vehicle to move in a direction indicated by the arrow.
 5. The system of claim 4, wherein the user interface includes selectable curved arrows enabling the operator to pivot and pan the remote vehicle and selectable arrows that enable the operator to drive the remote vehicle forward and backward.
 6. The system of clam 1, wherein the remote vehicle further comprises a manipulator arm and the operator control unit provides an operator with a virtualized 3D view of the pose of the manipulator arm.
 7. The system of claim 6, wherein instructions for moving the manipulator arm are sent from the operator control unit.
 8. The system of claim 6, wherein the manipulator arm further comprises a gripper.
 9. The system of claim 6, wherein the operator control unit further comprises a point-and-click interface on the 3D display of the manipulator arm such that selecting the manipulator arm on the 3D map display and dragging it to a desired pose or position causes the remote vehicle to execute the motions required to achieve the desired pose or position.
 10. The system of claim 1, further comprising a computational payload, the computational payload converting the raw data to a 3D map display on the operator control unit by constructing a virtual view of the 3D data directly from the imaging device to create an interactive display.
 11. The system of claim 1, wherein the interactive 3D map display includes a local obstacle map constructed using range data provided by the imaging device. 